Skip to content

Draft: Feat/backend go#9

Merged
samcm merged 52 commits intomasterfrom
feat/backend-go
Apr 22, 2025
Merged

Draft: Feat/backend go#9
samcm merged 52 commits intomasterfrom
feat/backend-go

Conversation

@samcm
Copy link
Copy Markdown
Member

@samcm samcm commented Mar 27, 2025

No description provided.

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages bot commented Mar 27, 2025

Deploying lab with  Cloudflare Pages  Cloudflare Pages

Latest commit: 1bc6e20
Status:🚫  Build failed.

View logs

@parithosh
Copy link
Copy Markdown
Member

LGTM

samcm added 10 commits April 8, 2025 15:01
This commit introduces a new script, `create_new_proto_file.sh`,
that automates the creation of new protocol buffer files and
directories. The script takes a proto name as an argument, creates
the corresponding directory and `.proto` file, and updates the
`buf.work.yaml` file to include the new directory. This streamlines
the process of adding new proto definitions to the project.
feat: add storage compression
feat: add storage store params and unified store method
feat: add xatu public contributors service

This commit introduces several enhancements and fixes:

- Adds integration tests for ClickHouse client using testcontainers.
- Implements storage compression using gzip algorithm.
- Introduces StoreParams struct and a unified Store method in the storage
  package to handle encoding, compression, atomicity, content type, and
  metadata.
- Implements the Xatu Public Contributors service.
- Fixes and improves the beacon chain timings service.
- Updates the beacon slots service.
- Updates the xatu public contributors service.
- Adds .gitignore entry for debug_bin
samcm added 25 commits April 15, 2025 14:32
This commit refactors the storage client to accept a context for
cancellation and deadlines. It also adds compression to the stored
data to reduce storage costs and improve performance.

The following changes were made:

- Updated the `storage.Get` and `storage.Store` methods to accept a
  context.
- Added compression to the `storage.Store` method.
- Updated all services to use the new `storage.Get` and
  `storage.Store` methods.
- Removed bucket creation logic from storage client and fail if bucket
  does not exist.
- Removed content type detection logic from storage client and rely on
  codec.
feat(beacon_slots.go): implement beacon slots gRPC service
feat(beacon_slots.go): implement beacon slots service
feat: add data fetching logic for beacon slots

This commit introduces the beacon slots service, including its
configuration, gRPC handler, and core logic. It also includes
data fetching from ClickHouse and transformation for storage.
The changes include:

- Added `modules` config to `config.go`
- Implemented the `BeaconSlotsHandler` in `grpc/beacon_slots.go`
- Implemented the `BeaconSlots` service in `internal/service/beacon_slots/beacon_slots.go`
- Added data fetching logic in `internal/service/beacon_slots/data_fetching.go`
- Added transformation logic in `internal/service/beacon_slots/transform.go`
- Added processing logic in `internal/service/beacon_slots/processing.go`
- Removed `json.go`

feat: add BlockArrivalTime, BlobArrivalTime, BlobArrivalTimes messages
feat: add FullTimings message to include metadata and geo location
feat: add SlimTimings message to drop metadata and geo location
feat: update BeaconSlotData message to use SlimTimings
feat: add BeaconSlots service definition
refactor: remove BeaconSlotsService and related code
refactor: rename Timings to SlimTimings
refactor: update beacon_slots.proto to include new messages
feat(go.mod): add ethwallclock dependency
feat(go.mod): add prysmaticlabs/go-bitfield dependency
feat(go.mod): remove durafmt dependency
feat(pkg/internal/lab/ethereum): add config struct for ethereum
feat(pkg/internal/lab/ethereum): add network struct for ethereum
feat(pkg/internal/lab/state): add state management package
feat(pkg/internal/lab/storage): add exists method to storage client
feat(pkg/server/internal/service/beacon_chain_timings): refactor state
feat(pkg/server/internal/service/beacon_chain_timings): add state adapter
feat(pkg/server/internal/service/beacon_slots): refactor beacon slots
feat(pkg/server): refactor server to use new ethereum client
feat: remove xatuclickhouse generated files

This commit removes the generated files for the xatuclickhouse package.
The processing logic was changed to be done in the server side.

feat(processing.go): process slots data in parallel

This commit introduces parallel processing of slot data to reduce
latency. It uses errgroup to run independent data fetching tasks
concurrently, improving overall performance.
```

chore: remove unused xatuclickhouse models

This commit removes a large number of unused models from the
`pkg/xatuclickhouse/models` directory. These models were autogenerated
but are no longer used in the project. Removing them cleans up the
codebase and reduces the potential for confusion.

feat: remove mev_relay_validator_registration model and ptr utils

The mev_relay_validator_registration model and ptr utils are removed
because they are no longer needed.
This commit significantly updates the README to accurately
describe the current Go-based backend architecture, replacing
the outdated Python implementation details. It includes
details on the SRV and API services, key backend modules,
technologies used, and updated development instructions.
The multi-network support feature description is also updated.
feat: introduce config API and related data structures

This commit introduces the config API, which allows retrieving
configuration information for the lab. The commit also defines
the data structures used by the config API, including:

- ConfigResponse: The main response message for the config API.
- ModuleConfig: Configuration for a specific module.
- TimeWindow: A time window for a module.
- EthereumConfig: Configuration for Ethereum.
- EthNetworkConfig: Configuration for an Ethereum network.
- Forks: Fork configuration for an Ethereum network.
- ForkDetail: Details about a specific fork.

build: regenerate lab_api.pb.go

This commit updates the lab_api.pb.go file by regenerating it.
This ensures that the generated code is up-to-date with the
latest proto definition.

feat: add GetFrontendConfig endpoint to lab API

This commit introduces a new `GetFrontendConfig` endpoint to the
Lab API. This endpoint retrieves the frontend configuration,
allowing clients to dynamically adjust their behavior based on
the server's configuration.

The changes include:
- Added the `GetFrontendConfig` method to the `LabService` in
  `pkg/server/internal/grpc/lab.go`.
- Implemented the `GetFrontendConfig` method in the
  `pkg/api/service.go` to handle the new endpoint.
- Added the corresponding protobuf definitions in
  `pkg/api/proto/lab_api.proto` and updated the generated code.
- Modified the `pkg/server/internal/service/lab/lab.go` to
  implement the logic for retrieving the frontend configuration.
- Updated the `pkg/server/internal/service/beacon_slots/beacon_slots.go`
  to include the `FrontendModuleConfig` method.
- Updated the `pkg/server/internal/service/beacon_chain_timings/beacon_chain_timings.go`
  to include the `FrontendModuleConfig` method.
- Updated the `pkg/server/internal/service/xatu_public_contributors/xatu_public_contributors.go`
  to include the `FrontendModuleConfig` method.
- Removed the `pkg/server/internal/service/lab.go` file.

feat(proto): add GetConfigResponse message and module config
feat(server): add lab service and update dependencies

This commit introduces a new GetConfigResponse message in the
protocol buffer definition, which includes module configurations
and Ethereum settings. It also adds a new lab service that
depends on the Ethereum client, cache client, beacon chain
timings, Xatu public contributors, and beacon slots services.
The server initialization logic is updated to include the new
lab service. The stop function is updated to stop all services
gracefully.
This commit improves the service's shutdown process by using a
cancelable context and WaitGroup to ensure all components stop
gracefully. It also adds a timeout to prevent indefinite blocking
during shutdown and logs warnings if the timeout is exceeded.
Additionally, the commit passes the service context to dependency
initialization functions.
This change ensures that the process context is derived from the
context passed to the Start method, allowing for better control and
propagation of cancellation signals.
The backend directory and its contents have been removed. This includes the Dockerfile, configuration files, Python scripts, and related modules. The removal is part of a larger refactoring effort to streamline the project and potentially integrate backend functionalities elsewhere or rely on external services.

feat: add metrics instrumentation to cache, clickhouse, ethereum, and geolocation services

This commit introduces metrics instrumentation to the cache, clickhouse, ethereum, and geolocation services using the prometheus client library. The changes include:

- Added metrics collection for cache operations (get, set, delete) with hit/miss/error status, operation duration, and number of items in the cache.
- Added metrics collection for ClickHouse queries (query, query_row, exec) with success/error status and query duration.
- Added metrics collection for Ethereum network wallclock and configuration loading.
- Added metrics collection for geolocation lookups with success/error/not_found status, lookup duration, and database load operations.

The metrics are exposed through Prometheus and can be used to monitor the performance and health of the services.

feat(leader): add metrics to leader election process
feat(geolocation): add metrics service to geolocation
feat(locker): add metrics to distributed lock implementation
feat(state): add metrics to state management
feat(storage): add metrics to storage operations
feat(xatu): add metrics to xatu client
feat(beacon_chain_timings): add metrics to beacon chain timings service
feat(beacon_slots): add metrics to beacon slots service
feat(lab): add metrics to lab service
feat(xatu_public_contributors): add metrics to xatu public contributors service

feat: add metrics endpoint and HTTP server

This commit introduces a new HTTP server that exposes Prometheus
metrics at the `/metrics` endpoint. The server runs on port 9090 by
default.

The following changes were made:

- Added a new `httpServer` field to the `Service` struct.
- Added a new `startMetricsServer` method to start the HTTP server.
- Modified the `Start` method to start the HTTP server.
- Modified the `stop` method to gracefully shut down the HTTP server.
- Initialized metrics service as a dependency.
- Passed metrics service to all services.
The metricsCollector is initialized during service startup. Therefore, nil checks before using the collector are unnecessary and have been removed to simplify the code.
Introduces new Prometheus metrics to track the state and age of processed data within the `beacon_chain_timings`, `beacon_slots`, and `xatu_public_contributors` services.

- `beacon_chain_timings`: Adds `state_last_processed_seconds` and `state_age_seconds` to track the last processed time and age of block timings and CDF data per network and window.
- `beacon_slots`: Adds `state_last_processed_slot` and `state_slot_age` to track the last processed slot and its age (in slots) for each processor (head, trailing, backfill) per network.
- `xatu_public_contributors`: Adds `state_last_processed_seconds`, `state_window_last_processed_seconds`, `state_age_seconds`, and `state_window_age_seconds` to track the last processed time and age for each processor and its specific time windows per network.

These metrics provide better visibility into the freshness and processing status of the data managed by these services.
…ocessing

fix(beacon_slots): increase leader refresh interval to reduce etcd load
feat(api): enable xatu_public_contributors module by default
fix(frontend): update vite proxy target port to 8080
…upport flat and nested formats

refactor(beacon_slots): update SQL query to include geo and time data for block events
feat(geolocation): add support for 'coordinates' header in CSV download
fix(ci): update component name from backend to pkg in containers workflow
@samcm samcm merged commit 34d6648 into master Apr 22, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants